<!DOCTYPE html>
<html>
  
<head><meta name="generator" content="Hexo 3.8.0">
  <meta charset="utf-8">
  <meta name="author" content="Amos Zhu">
  
  
  <title>Kafka架构探险（三）——Kafka参数详解 | Amos的技术博客</title>

  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">

  
    <meta name="keywords" content="kafka,kafka,">
  

  
  <meta name="description" content="Kafka提供很多的参数：broker参数、topic参数、producer参数、consumer参数等，方便我们对kafka的性能做调整，本博文就来归纳总结下常见的参数">

  

  
    <script src="//cdn.jsdelivr.net/npm/leancloud-storage@3.11.1/dist/av-min.js" async></script>
  

  
    <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-MML-AM_CHTML" async></script>
  

  
    <script src="//unpkg.com/valine/dist/Valine.min.js" async></script>
  

  

  <script>
  // theme-ad's config script
  // it can be used in every script
  
  window.AD_CONFIG = {
    leancloud: {"appid":"eObxCkTJ6j4GzRxGLgiHyfJa-gzGzoHsz","appkey":"fzRMAFC6MCE11mp7mnvh4Kwe","comment":true,"count":true},
    welcome: {"enable":true,"interval":30},
    start_time: "2019-05-28",
    passwords: ["efe07af7441da2b69c4a41e42e73be4db47f66010a56900788a458354a7373ec", ],
    is_post: true,
    lock: false,
    author: "Amos Zhu",
    share: {"qq":true,"wechat":true},
    mathjax: true,
    page_type: "",
    root: "/blog/"
  };
</script>

  <script src="/blog/vendor/sha256.min.js"></script>
<script src="/blog/js/auth.js"></script>
<script src="/blog/js/index.js"></script>
<script src="/blog/vendor/qrcode.min.js"></script>

  
    <link rel="icon" href="/blog/images/favicon.ico">
    <link rel="apple-touch-icon" href="/blog/images/touch-icon.png">
  

  <link href="//netdna.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css" rel="stylesheet">

  <link rel="stylesheet" href="/blog/css/index.css">
<link rel="stylesheet" href="/blog/styles/components/highlight/highlight.css">

  
</head>
  <body>
    <header class="site-header">
  <div class="site-header-brand">
    
      <span class="site-header-brand-title">
        <a href="/blog/">Amos Zhu</a>
      </span>
    
    
      <span class="site-header-brand-motto"> | 当浮躁的心静不下，不妨写些东西或者看会书</span>
    
  </div>
  <div class="site-header-right">
    <nav class="site-header-navigation">
      
        <a href="/blog/" target="_self">首页</a>
      
        <a href="/blog/archives/" target="_self">归档</a>
      
        <a href="/blog/tags/" target="_self">标签</a>
      
        <a href="/blog/categories/" target="_self">分类</a>
      
        <a href="/blog/redis/" target="_self">redis</a>
      
        <a href="/blog/rabbitMQ/" target="_self">rabbitMQ</a>
      
        <a href="/blog/elasticsearch" target="_self">elasticsearch</a>
      
        <a href="/blog/stuff" target="_self">stuff</a>
      
        <a href="/blog/kafka" target="_self">kafka</a>
      
        <a href="/blog/friends/" target="_self">友链</a>
      
        <a href="/blog/about/" target="_self">关于</a>
      
    </nav>
    <div class="site-header-btn">
      
        <a href="https://gitee.com/amos_zhu" target="_blank" id="site-github">
          <i class="fa fa-github-alt"></i>
        </a>
      
      <a href="javascript:void(0);" id="site-search">
        <i class="fa fa-search"></i>
      </a>
      <a href="javascript:void(0);" id="site-nav-btn">
        <i class="fa fa-ellipsis-v"></i>
      </a>
    </div>
  </div>
</header>
<nav class="table-content" id="site-nav">
  <div class="table-content-title">
    <span>导航</span>
  </div>
  <div class="table-content-main">
    <ol class="toc">
      
        <li class="toc-item">
          <a href="/blog/" target="_self">
            首页
          </a>
        </li>
      
        <li class="toc-item">
          <a href="/blog/archives/" target="_self">
            归档
          </a>
        </li>
      
        <li class="toc-item">
          <a href="/blog/tags/" target="_self">
            标签
          </a>
        </li>
      
        <li class="toc-item">
          <a href="/blog/categories/" target="_self">
            分类
          </a>
        </li>
      
        <li class="toc-item">
          <a href="/blog/redis/" target="_self">
            redis
          </a>
        </li>
      
        <li class="toc-item">
          <a href="/blog/rabbitMQ/" target="_self">
            rabbitMQ
          </a>
        </li>
      
        <li class="toc-item">
          <a href="/blog/elasticsearch" target="_self">
            elasticsearch
          </a>
        </li>
      
        <li class="toc-item">
          <a href="/blog/stuff" target="_self">
            stuff
          </a>
        </li>
      
        <li class="toc-item">
          <a href="/blog/kafka" target="_self">
            kafka
          </a>
        </li>
      
        <li class="toc-item">
          <a href="/blog/friends/" target="_self">
            友链
          </a>
        </li>
      
        <li class="toc-item">
          <a href="/blog/about/" target="_self">
            关于
          </a>
        </li>
      
    </ol>
  </div>
</nav>
<div id="site-process"></div>
    <main>
      
  <div class="passage">
  <div class="passage-meta">
    <span>
      <i class="fa fa-calendar"></i>2019-10-18
    </span>
    
      <span>
        | <a href="/blog/categories/kafka/"><i class="fa fa-bookmark"></i>kafka</a>
      </span>
    
    
      <span>
        | <i class="fa fa-unlock-alt"></i>UNLOCK
      </span>
    
  </div>
  <h1 class="passage-title">
    Kafka架构探险（三）——Kafka参数详解
  </h1>
  
  <article class="passage-article">
    <br><strong>参数的说明是基于Kafka 2.12-2.3.0版本</strong><p></p>

<p>详细可以参考<a href="http://kafka.apache.org/documentation.html#producerapi" target="_blank" rel="noopener">官网</a></p>
<h3 id="broker参数"><a href="#broker参数" class="headerlink" title="broker参数"></a>broker参数</h3><p>broker端的参数在Kafka目录下的<code>config/server.properties</code>文件中</p>
<table>
<thead>
<tr>
<th>参数名称</th>
<th>备注说明</th>
</tr>
</thead>
<tbody>
<tr>
<td>broker.id</td>
<td>kafka的唯一标志，全局唯一，默认是-1，如果不指定，则自动生成，也是全局唯一</td>
</tr>
<tr>
<td>log.dirs</td>
<td>指定了kafka持久化消息的目录；可以是多个值，以逗号隔开；设置多个目录，可以提高系统的吞吐量</td>
</tr>
<tr>
<td>zookeeper.connect</td>
<td>kafka依赖的zookeeper的ip和port；可以是zookeeper集群，以逗号隔开</td>
</tr>
<tr>
<td>listeners</td>
<td>broker监听器，主要用于client连接broker使用，格式：协议://ip:port,协议://ip:port,协议包括：PLAINTEXT/SSL/SASL_SSL等，如果不指定，则默认所有的client都能连接broker</td>
</tr>
<tr>
<td>advertised.listeners</td>
<td>和listeners类似，主要用于laas环境下，在多网卡的环境中，可以设置该参数值提供给外网的client访问</td>
</tr>
<tr>
<td>log.retention.{hours/minutes/ms}</td>
<td>控制消息数据留存的时间,同时设置优先ms，默认留存时间是7d</td>
</tr>
<tr>
<td>log.retention.bytes</td>
<td>控制消息日志保存的大小，超过该值的大小，则kafka会自动清理该分区的过期日志端文件，默认值是1073741824，值为-1时，表示不会根据消息的大小来删除消息日志</td>
</tr>
<tr>
<td>num.network.threads</td>
<td>控制broker端在后台用于处理网络请求的线程数，默认是3</td>
</tr>
<tr>
<td>num.io.threads</td>
<td>控制broker端处理磁盘IO的线程数，默认是8</td>
</tr>
<tr>
<td>message.max.bytes</td>
<td>kafka接收最大消息的大小，默认是977kb</td>
</tr>
<tr>
<td>log.segment.bytes</td>
<td>topic的分区是以一堆segment文件存储的，这个控制每个segment的大小，会被topic创建时的指定参数覆盖</td>
</tr>
<tr>
<td>log.retention.check.interval.ms</td>
<td>周期性检查是否有日志符合删除的条件,默认值是300s</td>
</tr>
<tr>
<td>num.recovery.threads.per.data.dir</td>
<td>kafka broker启动的过程中会加载此节点上所有topic的log文件，如果数据量非常大会导致加载时间过长,通过修改该值可以加快加载的速度，默认值是1</td>
</tr>
<tr>
<td>unclean.leader.election.enable</td>
<td>是否允许非ISR中的副本被选举为leader，如果设置为true的话，broker端可能因日志截断而造成消息丢失</td>
</tr>
<tr>
<td>replication.factor</td>
<td>设置主题的副本数，一般使用default.replication.factor</td>
</tr>
</tbody>
</table>
<h3 id="topic参数"><a href="#topic参数" class="headerlink" title="topic参数"></a>topic参数</h3><p>如果将broker参数看作是全局参数设置，那么topic参数就是局部参数设置，参数名称和broker中的参数名称是一致的，不过在topic中设置了，会覆盖broker中的参数设置，常见的有以下几种：<br>参数名称 | 备注说明<br>—|—<br>message.max.bytes | 覆盖全局的 message.max.bytes，即为每个topic指定不同的最大消息的大小<br>retention.bytes | 覆盖全局的log.retention.bytes 每个topic设置不同的消息留存大小</p>
<h3 id="producer参数"><a href="#producer参数" class="headerlink" title="producer参数"></a>producer参数</h3><p>消息生产者发送消息到kafka指定的topic中,所以producer端的参数大多跟topic和消息有关</p>
<table>
<thead>
<tr>
<th>参数名称</th>
<th>备注说明</th>
</tr>
</thead>
<tbody>
<tr>
<td>key.serializer</td>
<td>消息序列化，可以指定系统已经提供的序列化方式，也可以自定义序列化方式 ，自定义需要实现org.apache.kafka.common.serialization.Serializer接口，参数值是序列化处理器的全路径名</td>
</tr>
<tr>
<td>value.serializer</td>
<td>跟key.serializer一样</td>
</tr>
<tr>
<td>acks</td>
<td>控制producer生产消息的持久性，值范围有0、1、all</td>
</tr>
<tr>
<td>buffer.memory</td>
<td>指定producer端用于缓存消息的缓冲区大小，单位是字节，默认是32MB</td>
</tr>
<tr>
<td>compression.type</td>
<td>消息压缩的类型,默认是none,值范围：GZIP、Snappy、LZ4</td>
</tr>
<tr>
<td>retries</td>
<td>消息重试的次数</td>
</tr>
<tr>
<td>batch.size</td>
<td>producer端批量发送消息的批大小</td>
</tr>
<tr>
<td>request.timeout.ms</td>
<td>broker响应producer端的超时时间大小,默认是30s</td>
</tr>
<tr>
<td>max.block.ms</td>
<td>内存缓冲区在填满时，producer处于阻塞状态，并停止接收新的消息而不是抛出异常</td>
</tr>
<tr>
<td>max.in.flight.requests.per.connection</td>
<td>限制了producer在单个broker链接上能否发送的未响应请求的数量，如果设置成1，则可以保证消息的顺序性</td>
</tr>
</tbody>
</table>
<h3 id="consumer参数"><a href="#consumer参数" class="headerlink" title="consumer参数"></a>consumer参数</h3><table>
<thead>
<tr>
<th>参数名称</th>
<th>备注说明</th>
</tr>
</thead>
<tbody>
<tr>
<td>group.id</td>
<td>指定consumer group 的名字</td>
</tr>
<tr>
<td>key.deserializer</td>
<td>为消息的key反序列化，也可以自定义，需要实现org.apache.kafka.common.serialization.Deserializer接口</td>
</tr>
<tr>
<td>value.deserializer</td>
<td>为消息体反序列化，跟key.deserializer类似</td>
</tr>
<tr>
<td>session.timeout.ms</td>
<td>consumer group检测组内成员发送崩溃的时间(coordinator检测失败的时间)</td>
</tr>
<tr>
<td>max.poll.interval.ms</td>
<td>consumer处理逻辑最大的时间</td>
</tr>
<tr>
<td>auto.offset.reset</td>
<td>指定了无位移或位移移界（即consumer要消费的消息的位移不在当前消息日志的合理区间范围）时kafka的应对策略，有三个值：earliest、latest、none</td>
</tr>
<tr>
<td>enable.auto.commit</td>
<td>指定consumer是否自动提交位移</td>
</tr>
<tr>
<td>fetch.max.bytes</td>
<td>指定consumer端单次获取数据的最大字节数，若消息的内容很大，则该值应该调更大点，否则这个消息无法被消费,默认是52428800</td>
</tr>
<tr>
<td>max.poll.records</td>
<td>控制单次poll调用返回的最大消息数，默认是500条</td>
</tr>
<tr>
<td>heartbeat.interval.ms</td>
<td>coordinator要开启新一轮的rebalance时，该参数控制consumer group的成员何时得知新的rebalance，该值必须要小于session.timeout.ms</td>
</tr>
<tr>
<td>connections.max.idle.ms</td>
<td>Kafka定义关闭空闲Socket的时间间隔，默认是9分钟</td>
</tr>
</tbody>
</table>
  </article>
  <aside class="table-content" id="site-toc">
  <div class="table-content-title">
    <i class="fa fa-arrow-right fa-lg" id="site-toc-hide-btn"></i>
    <span>目录</span>
  </div>
  <div class="table-content-main">
    <ol class="toc"><li class="toc-item toc-level-3"><a class="toc-link" href="#broker参数"><span class="toc-text">broker参数</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#topic参数"><span class="toc-text">topic参数</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#producer参数"><span class="toc-text">producer参数</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#consumer参数"><span class="toc-text">consumer参数</span></a></li></ol>
  </div>
</aside>
  
    <aside class="passage-copyright">
      <div>本文作者: Amos Zhu</div>
      
        <div>
          原文链接: 
          <a href target="_blank">http://amos_zhu.gitee.io/passages/Kafka架构探险（三）——Kafka参数详解/</a>
        </div>
      
      <div>
        版权声明: 本博客所有文章除特别声明外, 均采用 <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">CC BY-NC-SA 4.0</a> 许可协议. 转载请注明出处!
      </div>
    </aside>
  
  
    <div class="passage-tags">
     
      <a href="/blog/tags/kafka/"><i class="fa fa-tags"></i>kafka</a>
    
    </div>
  
</div>

    </main>
    
      
<div class="site-comment-contanier" data-plateform="leancloud">
  
    <p id="site-comment-info">
      <i class="fa fa-spinner fa-spin"></i> 评论加载中
    </p>
    <div id="site-comment"></div>
  
</div>
    
    <div class="site-footer-wrapper">
  <footer class="site-footer">
    
      
        <div class="site-footer-col">
          <h5 class="site-footer-title">博客推荐</h5>
          
            <span class="site-footer-item">
              <a href="http://youzhixueyuan.com/series/java" target="_blank">优知学院</a>
            </span>
          
            <span class="site-footer-item">
              <a href="http://ruanyifeng.com/" target="_blank">阮一峰的个人网站</a>
            </span>
          
        </div>
      
        <div class="site-footer-col">
          <h5 class="site-footer-title">系列教程</h5>
          
            <span class="site-footer-item">
              <a href="http://amos_zhu.gitee.io/blog/redis" target="_blank">Redis系列教程</a>
            </span>
          
            <span class="site-footer-item">
              <a href="http://amos_zhu.gitee.io/blog/rabbitMQ/" target="_blank">RabbitMQ教程</a>
            </span>
          
            <span class="site-footer-item">
              <a href="http://amos_zhu.gitee.io/blog/kafka/" target="_blank">Kafka架构探险</a>
            </span>
          
        </div>
      
        <div class="site-footer-col">
          <h5 class="site-footer-title">抓到我</h5>
          
            <span class="site-footer-item">
              <a href="https://gitee.com/amos_zhu" target="_blank">Gitee</a>
            </span>
          
            <span class="site-footer-item">
              <a href="https://blog.csdn.net/wdcl2468" target="_blank">CSDN</a>
            </span>
          
            <span class="site-footer-item">
              <a href="https://www.jianshu.com/u/2dbe61b1f3c3" target="_blank">简书</a>
            </span>
          
        </div>
      
    
    <div class="site-footer-info">
      <i class="fa fa-clock-o"></i> 本站已稳定运行<span id="site-time"></span>
    </div>
    
      <div class="site-footer-info">
        <i class="fa fa-paw"></i> 您是本站第 <span id="site-count"></span> 位访客
      </div>
    
    
      <div class="site-footer-info">
        <i class="fa fa-at"></i> Email: amoszhu@aliyun.com
      </div>
    
    <div class="site-footer-info">
      <i class="fa fa-copyright"></i> 
      2019 <a href="https://github.com/dongyuanxin/theme-ad/" target="_blank">Theme-AD</a>.
      Created by <a href="https://godbmw.com/" target="_blank">GodBMW</a>.
      All rights reserved.
    </div>
  </footer>
</div>
    <div id="site-layer" style="display:none;">
  <div class="site-layer-content">
    <div class="site-layer-header">
      <span class="site-layer-header-title" id="site-layer-title"></span>
      <i class="fa fa-close" id="site-layer-close"></i>
    </div>
    <div class="site-layer-body" id="site-layer-container">
      <div class="site-layer-input" id="site-layer-search" style="display: none;">
        <div class="site-layer-input-choose">
          <a href="javascript:void(0);" title="Change Search Engine">Google</a>
        </div>
        <input type="text">
        <i class="fa fa-search"></i>
      </div>
      
        <div class="site-layer-reward" id="site-layer-reward" style="display: none;">
          
            <div>
              <img src="/blog/images/wechat.png" alt="WeChat">
              
                <p>WeChat</p>
              
            </div>
          
            <div>
              <img src="/blog/images/alipay.png" alt="AliPay">
              
                <p>AliPay</p>
              
            </div>
          
        </div>
      
      <div id="site-layer-welcome" style="display:none;"></div>
    </div>
  </div>
</div>
    

<div class="bottom-bar">
  <div class="bottom-bar-left">
    <a href="javascript:void(0);" data-enable="false">
      <i class="fa fa-arrow-left"></i>
    </a>
    <a href="/blog/passages/Kafka架构探险（二）——一文带你了解Kafka/" data-enable="true">
      <i class="fa fa-arrow-right"></i>
    </a>
  </div>
  <div class="bottom-bar-right">
    <a href="javascript:void(0);" data-enable="true" id="site-toc-show-btn">
      <i class="fa fa-bars"></i>
    </a>
    
      <a href="#site-comment" data-enable="true">
        <i class="fa fa-commenting"></i>
      </a>
    
    <a href="javascript:void(0);" id="site-toggle-share-btn">
      <i class="fa fa-share-alt"></i>
    </a>
    
      <a href="javascript:void(0);" id="site-reward">
        <i class="fa fa-thumbs-up"></i>
      </a>
    
    <a href="javascript:void(0);" id="back-top-btn">
      <i class="fa fa-chevron-up"></i>
    </a>
  </div>
</div>
    <div id="share-btn">
  
  
  
  
    <a id="share-btn-qq" href="javascript:void(0);" target="_blank">
      <i class="fa fa-qq"></i>
    </a>
  
  
    <a id="share-btn-wechat" href="javascript:void(0);" target="_blank">
      <i class="fa fa-wechat"></i>
    </a>
  
</div>
    


  <script async>
  (function(){
      var bp = document.createElement('script');
      var curProtocol = window.location.protocol.split(':')[0];
      if (curProtocol === 'https') {
          bp.src = 'https://zz.bdstatic.com/linksubmit/push.js';
      }
      else {
          bp.src = 'http://push.zhanzhang.baidu.com/push.js';
      }
      var s = document.getElementsByTagName("script")[0];
      s.parentNode.insertBefore(bp, s);
  })();
  </script>



  <script async>
    (function(){
    var src = (document.location.protocol == "http:") ? "http://js.passport.qihucdn.com/11.0.1.js?":"https://jspassport.ssl.qhimg.com/11.0.1.js?";
    document.write('<script src="' + src + '" id="sozz"><\/script>');
    })();
  </script>


    
  </body>
</html>